In theory baking maps and UVs is pretty straightforward, especially when it is working as planned. Despite that, some light reading is in order.
Baking Maps for Games Reading List
* ========================================== *
UV Mapping Beginner's Tips by RizomUV
In-depth with some nice specific settings info unrelated to RizomUV.
"If you have to stack and flip, make sure to use an FBX export with tangents and binormals checked."
Normal Map Specific
Height Maps vs Normal Maps (Video 03:22)
Illustrated Normal Tangent Space, Object Space, and World Space
Takeaways for my current Marmoset Toolbag 4 baking project with final render in Unity.
- Height Map - To ensure that your zero point is 50%, set the min and max values proportionally, for instance, -5 and 5
- Smooth the cage, get more skew, less crazy edges. Fix skew with skew paint.
- Try split uvs instead of continuous.
- Make sure to triangulate your mesh before importing it into Toolbag. (triangulation mismatch causes x-shaped smoothing errors )
- Right handed (OpenGL) for Unity (handedness can be overridden via the normal map settings in the Baker object by clicking the Flip Y checkbox )
- Dither Map
- 8-bit tga, 16-bit tiff (Unity alpha 2020.2.0a13 supports 16-bit formats as of June 2020)
Baking to a higher bit depth (16-bit) helps to remove banding, and down-converting to 8-bit with dithers (in Photoshop) to more effectively cram the values into the 8 bit space. Screen captures from Bit Depth and How Compression Affects Normal Maps